Apparatus to perform alignment to images, image processing method to perform alignment to images, and computer readable non-transitory memory to perform alignment to images

ABSTRACT

An apparatus includes at least one memory configured to store instructions, and at least one processor in communication with the at least one memory and configured to execute the instructions to detect a feature amount from each of a plurality of images different in in-focus position in an optical axis direction, and align the plurality of images by using the feature amount. The at least one processor further executes instructions to detect the feature amount by preferentially using a focusing area of each of the plurality of images.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a Continuation of U.S. application Ser. No.17/153,637, filed Jan. 20, 2021, which claims priority from JapanesePatent Application No. 2020-014082, filed Jan. 30, 2020, which is herebyincorporated by reference herein in its entirety.

BACKGROUND OF THE DISCLOSURE Field of the Disclosure

The aspect of the embodiments relates to an image processing apparatusthat combines a plurality of images different in in-focus position in anoptical axis direction.

Description of the Related Art

In a case where images of a plurality of objects that is largelydifferent in distance from a digital camera are captured or in a casewhere an image of an object long in a depth direction is captured, apart of the object can be in focus because of an insufficient depth offield. To solve such an issue, Japanese Patent Application Laid-Open No.2015-186088 discusses a focus stacking composition technique in which aplurality of images different in in-focus position is captured, afocusing area is extracted from each of the images, and the extractedfocusing areas are combined to generate one composite image in which theentire imaging area is in focus. To perform the focus stackingcomposition, the images are aligned because field angles of the imagesdifferent in in-focus position are finely different and due to theeffect of camera shake in the imaging. Japanese Patent ApplicationLaid-Open No. 2015-186088 discusses a technique for performing alignmentin which blocks are set in each of the images to calculate a movingamount by using the blocks.

When the image alignment is performed, in one embodiment, the blocks arearranged first in the focusing area of the image because a featureamount such as a motion vector calculated from the focusing area of theimage is high in accuracy.

However, those images captured for the focus stacking composition aredifferent in position of the focusing area. When the blocks are arrangedin the same manner, accuracy of the alignment may be deteriorated.

SUMMARY OF THE DISCLOSURE

According to an embodiment of the disclosure, an image processingapparatus includes at least one memory configured to store instructions,and at least one processor in communication with the at least one memoryand configured to execute the instructions to detect a feature amountfrom each of a plurality of images different in in-focus position in anoptical axis direction, and align the plurality of images by using thefeature amount. The at least one processor further executes instructionsto detect the feature amount by preferentially using a focusing area ofeach of the plurality of images.

Further features of the disclosure will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a configuration of a digitalcamera according to an exemplary embodiment of the disclosure.

FIG. 2 is a flowchart illustrating generation of a composite imageaccording to the exemplary embodiment of the disclosure.

FIG. 3 is a flowchart illustrating imaging according to the exemplaryembodiment of the disclosure.

FIG. 4 is a flowchart illustrating alignment according to a firstexemplary embodiment.

FIGS. 5A to 5D are diagrams illustrating arrangement of blocks accordingto the first exemplary embodiment.

FIGS. 6A to 6E are diagrams illustrating rearrangement of blocksaccording to the first exemplary embodiment.

FIG. 7 is a flowchart illustrating image composition according to theexemplary embodiment of the disclosure.

FIG. 8 is a flowchart illustrating alignment processing according to asecond exemplary embodiment.

FIG. 9 is a flowchart illustrating another example of the alignmentprocessing according to the second exemplary embodiment.

DESCRIPTION OF THE EMBODIMENTS

Exemplary embodiments of the disclosure are described in detail belowwith reference to accompanying drawings.

FIG. 1 is an example of a block diagram illustrating a configuration ofa digital camera as an image processing apparatus according to a firstexemplary embodiment. A digital camera 100 can capture a still image,record information on a focus position, calculate a contrast value, andcombine images. Further, the digital camera 100 can perform enlargementprocessing or reduction processing on an image captured and stored, oran image input from outside.

A control unit 101 is a signal processor such as a central processingunit (CPU) and a micro processing unit (MPU), and controls units of thedigital camera 100 while reading out a program previously stored in aread only memory (ROM) 105 described below. For example, as describedbelow, the control unit 101 instructs an imaging unit 104 describedbelow to start and end imaging. The control unit 101 also instructs animage processing unit 107 described below to perform image processingbased on the program stored in the ROM 105. An instruction from a useris input to the digital camera 100 by an operation unit 110 describedbelow, and is transmitted to each of the units of the digital camera 100via the control unit 101.

A driving unit 102 includes a motor, and mechanically operates anoptical system 103 described below under the instruction of the controlunit 101. For example, the driving unit 102 moves a position of a focuslens included in the optical system 103 to adjust a focal length of theoptical system 103 based on the instruction of the control unit 101.

The optical system 103 includes a zoom lens, the focus lens, and adiaphragm. The diaphragm is a mechanism for adjusting a quantity oflight to be transmitted. A focus position is changeable by changing thepositions of the lenses.

The imaging unit 104 is a photoelectric conversion device that performsphotoelectric conversion to convert an incident optical signal into anelectric signal. For example, a charge-coupled device (CCD) sensor or acomplementary metal-oxide semiconductor (CMOS) sensor is adoptable asthe imaging unit 104. The imaging unit 104 has a moving image capturingmode, and can capture a plurality of temporally continuous images asframes of a moving image.

The ROM 105 is a nonvolatile read-only memory as a recording medium, andstores parameters for operation of each of the units in addition to anoperation program for each of the units included in the digital camera100. A random access memory (RAM) 106 is a rewritable volatile memory,and is used as a temporary storage area of data output in the operationof each of the units included in the digital camera 100.

The image processing unit 107 performs various image processing such aswhite balance adjustment, color interpolation, and filtering on theimages output from the imaging unit 104 or data of an image signalrecorded in a built-in memory 109 described below. The image processingunit 107 also performs compression processing conforming to a standardsuch as Joint Photographic Experts Group (JPEG) on data of the imagescaptured by the imaging unit 104.

The image processing unit 107 includes an integrated circuit(application specific integrated circuit (ASIC)) in which circuits eachperforming specific processing are integrated. Alternatively, thecontrol unit 101 may perform a part or all of the functions of the imageprocessing unit 107 by performing processing based on the program readout from the ROM 105. In a case where the control unit 101 performs allof the functions of the image processing unit 107, the image processingunit 107 as hardware is unnecessary.

A display unit 108 is a liquid crystal display, an organicelectroluminescence (EL) display, or other type of display that displaysan image temporarily stored in the RAM 106, an image stored in thebuilt-in memory 109 described below, and a setting screen of the digitalcamera 100.

The built-in memory 109 stores the image captured by the imaging unit104, the image processed by the image processing unit 107, informationon the focus position in imaging, etc. A memory card or the like may beused in place of the built-in memory 109.

The operation unit 110 includes, for example, buttons, switches, keys,and a mode dial provided on the digital camera 100, and examples of theoperation unit 110 includes a touch panel also serving as the displayunit 108. The instruction from the user is transmitted to the controlunit 101 via the operation unit 110.

FIG. 2 is a flowchart illustrating generation of a composite imageaccording to the present exemplary embodiment. In step S201, the imagingunit 104 captures a plurality of images at respectively differentin-focus positions in an optical axis direction. In step S202, thecontrol unit 101 performs alignment on the plurality of images capturedby the imaging unit 104 in step S201. In step S203, the image processingunit 107 combines the aligned images to generate a composite imagehaving a deeper depth of field. In the following, each of the steps willbe described in detail.

FIG. 3 is a flowchart illustrating the imaging in step S201 according tothe present exemplary embodiment.

In step S301, the control unit 101 sets an in-focus position. Forexample, the user designates a focus position via the touch panel alsoserving as the display unit 108, and designates a plurality of in-focuspositions at equal intervals on a front side and a rear side, in theoptical axis direction, of an in-focus position corresponding to thedesignated focus position. At the same time, the control unit 101determines an imaging order of the set in-focus positions in order of adistance.

In step S302, the imaging unit 104 captures an image at an in-focusposition, at which imaging has not been performed and which is theearliest in the imaging order among the in-focus positions set in stepS301.

In step S303, the control unit 101 determines whether images have beencaptured at all of the in-focus positions set in step S301. In a casewhere images have been captured at all of the set in-focus positions(YES in step S303), the processing in the flowchart is ended. In a casewhere there is an in-focus position where the image has not beencaptured (NO in step S303), the processing returns to step S302.

In a case of a multi lens camera including a plurality of imaging units104, images may be simultaneously captured at the plurality of in-focuspositions set in step S301.

FIG. 4 is a flowchart illustrating the alignment in step S202 accordingto the present exemplary embodiment. In the present exemplaryembodiment, the control unit 101 detects a motion vector as a featureamount of each image, and performs the alignment by using the motionvector.

In step S401, the image processing unit 107 calculates a contrast valuefrom each of the images captured in step S201. The contrast valuecalculated at this time is used for rearrangement of blocks and imagecomposition described below.

As an example of a method of calculating the contrast value, forexample, the image processing unit 107 first calculates a luminancevalue Y by using the following equation (1) from color signals Sr, Sg,and Sb of each of pixels.

Y=0.299Sr+0.587Sg+0.114Sb  (1)

Next, a Sobel filter is applied to a matrix L of luminance values Y of3×3 pixels, and a contrast value I is calculated by using the followingequations (2) to (4).

$\begin{matrix}{I_{h} = {\begin{pmatrix}{- 1} & 0 & 1 \\{- 2} & 0 & 2 \\{- 1} & 0 & 1\end{pmatrix} \cdot L}} & (2)\end{matrix}$ $\begin{matrix}{I_{v} = {\begin{pmatrix}{- 1} & {- 2} & {- 1} \\0 & 0 & 0 \\1 & 2 & 1\end{pmatrix} \cdot L}} & (3)\end{matrix}$ $\begin{matrix}{I = \sqrt{I_{h}^{2} + I_{v}^{2}}} & (4)\end{matrix}$

The above-described method of calculating the contrast value is merelyan example, and for example, an edge detection filter such as aLaplacian filter, or a bandpass filter allowing frequencies in apredetermined band to pass therethrough may be used.

In step S402, the control unit 101 determines target images on which thealignment processing is to be performed. Typically, the control unit 101determines, from among the images captured in step S201, two temporallyadjacent images captured at respective times next to each other aredetermined as the target images of the alignment processing. Forexample, a first captured image and a second captured image are firstdetermined as the target images.

In step S403, the control unit 101 detects a focusing area of each ofthe target images. As a method for detecting a focusing area, forexample, information on a focusing area is recorded in imaging, and thefocusing area is extracted in step S403. Alternatively, there is anotherexample of the method for detecting a focusing area in which contrastvalues of all captured images are compared with each other, and an areahaving a contrast value larger than a contrast value of a correspondingarea in the other image may be determined as the focusing area. At thistime, an interval between the in-focus positions of the two targetimages is relatively small, so that it is expected that the focusingareas are largely overlapped.

In step S404, the image processing unit 107 arranges blocks in each ofthe target images.

FIGS. 5A to 5D are diagrams illustrating arrangement of the blocksaccording to the present exemplary embodiment. FIG. 5A illustrates anexample of the captured image. FIG. 5B illustrates an example ofarrangement of blocks in the image illustrated in FIG. 5A. In theexample illustrated in FIG. 5B, blocks are arranged on the entire areaof the image illustrated in FIG. 5A. FIG. 5C illustrates an example inwhich blocks are arranged in a part of the area of the image illustratedin FIG. 5A. In FIG. 5B, blocks 501 and search ranges 502 including therespective blocks 501 are illustrated. The search ranges 502 arearranged for detection of a motion vector described below, based on therespective blocks.

Although the numbers of arranged blocks are the same in the examplesillustrated in FIG. 5B and FIG. 5C, the number of blocks is not limitedthereto. The number of blocks may be changed based on a calculationfunction of the control unit 101. To accurately detect the motion vectordescribed below, however, blocks similarly in all of the images are tobe arranged.

Further, in FIG. 5B and FIG. 5C, the search ranges are arranged so asnot to be overlapped with one another. However, the arrangement of thesearch ranges is not limited thereto. The search ranges may be arrangedso as to be overlapped with one another as illustrated in FIG. 5D. InFIG. 5D, although the sizes of the blocks and the search ranges aresimilar to the sizes of the blocks and the search ranges in FIG. 5B, theblocks and the search ranges do not cover the entire area of the imageunlike FIG. 5B because the search ranges are overlapped with oneanother.

In step S405, the image processing unit 107 detects a motion vector fromeach of the blocks arranged in step S404. The control unit 101calculates, in each of the search ranges in one of the target images, acorrespondence point at which a sum of absolute difference (hereinafter,referred to as SAD) of luminance values with respect to thecorresponding block of the other image becomes minimum. The control unit101 calculates the motion vector based on the center of the block andthe correspondence point of the center of the block. In theabove-described calculation of the correspondence point, the controlunit 101 may also use a sum of squared difference (hereinafter, referredto as SSD), normalized cross correlation (hereinafter, referred to asNCC), etc., in addition to the SAD.

In step S406, the image processing unit 107 determines whether themotion vectors detected in step S405 satisfy a predetermined condition.

To perform alignment with high accuracy, the motion vectors are requiredto satisfy a certain condition. For example, as described below, atleast three motion vectors not parallel to one another are applied tocalculate an affine transformation coefficient. In other words, thealignment cannot be performed unless a predetermined number or more ofmotion vectors are detected. Further, in a case where the detectedmotion vectors are excessively small and an influence of the differenceis large, the accuracy of the alignment may be low. In other words, thealignment cannot be performed unless the detected motion vectors areeach greater than or equal to a threshold value. Moreover, in a casewhere the motion vectors detected from the blocks are not directed to acertain direction, i.e., in a case where the deviation is large, theaccuracy of the alignment may be low. The image processing unit 107previously determines a condition that the motion vectors are tosatisfy, and the image processing unit 107 determines, in step S406,whether the motion vectors satisfy the above-described condition. In acase where the motion vectors do not satisfy the above-describedpredetermined condition (NO in step S406), the accuracy of the alignmentmay be low even if the alignment is performed. Therefore, the processingproceeds to step S407 without directly proceeding to the alignmentprocessing. In contrast, in a case where the detected motion vectorssatisfy the predetermined condition (YES in step S406), the processingproceeds to step S409.

In step S407, the image processing unit 107 re-arranges the blocks ineach of the target images with reference to the focusing area detectedin step S403. In one embodiment, the image processing unit 107 refers toa portion where the focusing areas of the two respective target imagesare overlapped with each other, and arranges the blocks at positionscorresponding to the overlapped portion.

FIGS. 6A to 6E are diagrams illustrating rearrangement of the blocksaccording to the present exemplary embodiment. An image 610 in FIG. 6Aand an image 620 in FIG. 6B are the two target images determined by thecontrol unit 101 in step S402. An area 611 is a focusing area of theimage 610 a, and an area 621 is a focusing area of the image 620. FIG.6C illustrates a state where the image 610 and the image 620 areoverlapped with each other. In FIG. 6C, a state where blocks arearranged in an area 631 where the focusing area 611 of the image 610 andthe focusing area 621 of the image 620 are overlapped is illustrated.FIG. 6D illustrates an area where the blocks are re-arranged in theimage 610, and FIG. 6E illustrates an area where the blocks arere-arranged in the image 620.

In the example of FIG. 6C, although the blocks are arranged over theentire area 631, the arrangement of the blocks is not limited thereto,and the blocks may be arranged in a part of the area 631.

If the control nit 101 determines a block rearrangement area, thecontrol unit 101 re-arranges the blocks in the block rearrangement areawith density higher than density of the blocks arranged in step S404.For example, the sizes of the blocks and the search ranges are madesmaller, and the same number of blocks are re-arranged. In a case wherethe search ranges arranged in step S404 are not overlapped with oneanother, the sizes of the blocks and the search ranges are not changed,and the blocks are re-arranged such that the search ranges of the blocksare overlapped with one another as illustrated in FIG. 5D. On the otherhand, in a case where the search ranges arranged in step S404 areoverlapped with one another, the sizes of the blocks and the searchranges may not be changed, and the blocks may be re-arranged such thatthe overlapped portions become larger.

In step S408, the image processing unit 107 re-detects the motion vectorfrom each of the blocks re-arranged in step S407. The method ofre-detecting a motion vector may be similar to the method of detectingthe motion vector in step S405.

In step S409, the control unit 101 determines whether the motion vectorsof all of the images have been calculated. In a case where there is animage in which a motion vector has not been calculated (NO in stepS409), the processing returns to step S402, and the control unit 101determines target images again. For example, the control unit 101determines two target images in order of imaging. In other words, in thefirst motion vector calculation s, the first image and the second imageare determined as target images. In second motion vector calculation,the second image and a third image are determined as target images.

If the motion vectors of all of the images are calculated (YES in stepS409), the processing proceeds to step S410. In step S410, a conversioncoefficient is calculated. The conversion coefficient is a coefficientfor deformation of an image in next step S411. For example, an image canbe deformed by using a conversion coefficient A expressed in thefollowing equation (5).

$\begin{matrix}{I^{\prime} = {\begin{pmatrix}x^{\prime} \\y^{\prime} \\1\end{pmatrix} = {{AI} = {\begin{pmatrix}a & b & c \\d & e & f \\g & h & i\end{pmatrix}\begin{pmatrix}x \\y \\1\end{pmatrix}}}}} & (5)\end{matrix}$

In the equation (5), (x′, y′) indicates coordinates after deformation,and (x, y) indicates coordinates before the deformation. Whencoordinates at a start point and an end point of each of the motionvectors detected from the two target images are respectively defined as(x, y) and (x′, y′) and three motion vectors not parallel to one anotherhave been detected, the conversion coefficient A in the equation (5) canbe calculated. Next, in step S411, the equation (5) is applied to allcoordinates in one of the target images by using the calculatedconversion coefficient. As a result, the two target images can bealigned. Applying such a method to all pairs of the target imagesdetermined in step S402 allows for alignment of all of the images.

Next, in step S203, the image processing unit 107 performs imagecomposition on the aligned images.

FIG. 7 is a flowchart illustrating image composition according to thepresent exemplary embodiment.

In step S701, the image processing unit 107 calculates contrast valuesof the respective aligned images. The contrast values can be calculatedby a method similar to the method described above in step S401. Further,the contrast value calculated in step S401 may be converted incoordinate in consideration of deformation, and the converted contrastvalue may be used.

In step S702, the image processing unit 107 generates a composition map.In a method of generating the composition map, the image processing unit107 compares the contrast values of pixels located at the same positionin the images, and calculates a composition ratio corresponding to themagnitude of the contrast values. More specifically, among the pixelslocated at the same position, the composition ratio of 100% is given tothe pixel having the largest contrast value, and the composition ratioof 0% is given to the other pixel located at the same position. In otherwords, the following equation (6) can be obtained.

$\begin{matrix}{{A_{m}( {x,y} )} = {\max\limits_{k = 1}{C_{k}( {x,y} )}}} & (6)\end{matrix}$

In the equation (6), C_(k)(x, y) indicates the contrast value calculatedin step S701, and A_(m)(x, y) indicates the ratio of the compositionmap. Further, m indicates an m-th image among the plurality of imagesdifferent in in-focus position, x indicates a horizontal coordinate ofthe image, and y indicates a vertical coordinate of the image.

In step S702, the composition ratio is appropriately adjusted to preventa boundary portion from becoming unnatural. As a result, the compositionratio of the composition map in one image is not binarized to 0% and100% but continuously changes.

In step S703, the image processing unit 107 generates a composite imagebased on the composition map. The generated composite image has a deeperdepth of field than the depth of field of each of the captured imagesbecause the composite image is generated by extracting the focusingareas in the captured images.

The above-described exemplary embodiment is merely illustrative, andvarious modifications can be implemented. For example, the followingmodifications can be made.

In the example described above, in the case where the detected motionvectors satisfy the predetermined condition in step S406, the conversioncoefficient is calculated in step S410 by using all of the detectedmotion vectors. The calculation, however, is not limited thereto. Forexample, even in the case where the detected motion vectors satisfy thepredetermined condition, in one embodiment, the control unit 101 usesthe motion vectors from the focusing area detected in step S403 in thecalculation of the conversion coefficient in step S410.

According to the first exemplary embodiment, in a case where such motionvectors that satisfy the predetermined condition cannot be calculated,the blocks are re-arranged in the focusing area, which allows for thealignment with higher accuracy.

A second exemplary embodiment is described below with reference to FIG.8 . Unlike the first exemplary embodiment, the blocks are arranged firstbased on the focusing area in the second exemplary embodiment. In thefollowing, the present exemplary embodiment will be described byfocusing on differences from the first exemplary embodiment.

A processing flow of the focus stacking composition according to thepresent exemplary embodiment is similar to the processing flow in FIG. 2according to the first exemplary embodiment. Processing flows of theimaging and the image composition are also similar to the processingflows according to the first exemplary embodiment.

FIG. 8 is a flowchart illustrating alignment processing according to thepresent exemplary embodiment. Processing in steps S801 to S803 issimilar to the processing in steps S401 to S403 according to the firstexemplary embodiment.

In step S804, the control unit 101 arranges blocks based on the focusingarea detected in step S803. A specific arrangement method may be, forexample, the method described with reference to FIGS. 6A to 6E in thefirst exemplary embodiment.

In step S805, the control unit 101 detects a motion vector of each ofthe target images based on the blocks arranged in step S804.

In step S809, the control unit 101 determines whether motion vectorshave been detected from all of the images. In a case where motionvectors have been detected from all of the images (YES in step S809),the processing proceeds to step S810. In a case where there is an imagefrom which a motion vector has not been detected (NO in step S809), theprocessing returns to step S802.

Processing in steps S810 and S811 is similar to the processing in stepsS410 and S411 according to the first exemplary embodiment.

Further, as a modification of the present exemplary embodiment, thefollowing implementation method is applicable. FIG. 9 is a flowchartillustrating a modification of the alignment processing according to thepresent exemplary embodiment. In the example illustrated in FIG. 8 ,after the motion vector is detected in step S805, the processingproceeds to the step of calculating the conversion coefficient. In theexample illustrated in FIG. 9 , the processing includes a step ofdetermining whether the motion vectors satisfy the predeterminedcondition. In other words, in the example illustrated in FIG. 9 , it isdetermined whether the detected motion vectors satisfy the predeterminedcondition in a manner similar to step S406 in the first exemplaryembodiment. In a case where the detected motion vectors do not satisfythe predetermined condition (NO in step S906), the blocks arere-arranged in step S907. In the present exemplary embodiment, since theblocks are arranged in consideration of the focusing area in step S904,in one embodiment, the blocks are arranged with higher density in therearrangement in step S907.

According to the second exemplary embodiment, a detection accuracy of amotion vector can be improved by arrangement of blocks used fordetection of the motion vector based on the position of a focusing area.

Other Exemplary Embodiments

Although the above-described exemplary embodiments have been describedbased on implementation in a digital camera, the implementation is notlimited to the digital camera. For example, the exemplary embodimentsmay be implemented in a mobile terminal with a built-in imaging device,or in a network camera that is capable of capturing an image.

The exemplary embodiments of the disclosure can be realized by supplyinga program for realizing one or more functions of the exemplaryembodiments to a system or an apparatus via a network or a storagemedium and causing one or more processors of a computer of the system orthe apparatus to read out and execute the program. Further, theexemplary embodiments of the disclosure can be realized by a circuit(e.g., ASIC) capable of realizing one or more functions.

Further features of the disclosure will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

Embodiment(s) of the disclosure can also be realized by a computer of asystem or apparatus that reads out and executes computer executableinstructions (e.g., one or more programs) recorded on a storage medium(which may also be referred to more fully as a ‘non-transitorycomputer-readable storage medium’) to perform the functions of one ormore of the above-described embodiment(s) and/or that includes one ormore circuits (e.g., application specific integrated circuit (ASIC)) forperforming the functions of one or more of the above-describedembodiment(s), and by a method performed by the computer of the systemor apparatus by, for example, reading out and executing the computerexecutable instructions from the storage medium to perform the functionsof one or more of the above-described embodiment(s) and/or controllingthe one or more circuits to perform the functions of one or more of theabove-described embodiment(s). The computer may comprise one or moreprocessors (e.g., central processing unit (CPU), micro processing unit(MPU)) and may include a network of separate computers or separateprocessors to read out and execute the computer executable instructions.The computer executable instructions may be provided to the computer,for example, from a network or the storage medium. The storage mediummay include, for example, one or more of a hard disk, a random-accessmemory (RAM), a read only memory (ROM), a storage of distributedcomputing systems, an optical disk (such as a compact disc (CD), digitalversatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, amemory card, and the like.

While the disclosure has been described with reference to exemplaryembodiments, it is to be understood that the disclosure is not limitedto the disclosed exemplary embodiments. The scope of the followingclaims is to be accorded the broadest interpretation so as to encompassall such modifications and equivalent structures and functions.

What is claimed is:
 1. An apparatus, comprising: at least one memoryconfigured to store instructions; and at least one processor incommunication with the at least one memory and configured to execute theinstructions to: arrange a region in each of a plurality of imagesdifferent in in-focus position in an optical axis direction; detect afeature amount from the region in each of the plurality of images; andalign the plurality of images by using the feature amount, and in a casewhere the detected feature amount does not satisfy a predeterminedcondition, re-arrange the region.
 2. The apparatus according to claim 1,wherein the at least one processor further executes instructions togenerate a composite image by extracting respective focusing areas ofthe plurality of images.
 3. The apparatus according to claim 2, whereinthe composite image has a depth of field deeper than a depth of field ofeach of the plurality of images.
 4. The apparatus according to claim 1,wherein the feature amount is a motion vector.
 5. The apparatusaccording to claim 1, wherein the at least one processor furtherexecutes instructions to arrange the region based on a focusing area ofeach of the plurality of images.
 6. The apparatus according to claim 2,wherein the at least one processor further executes instructions todetect the focusing area of each of the plurality of images by usingcontrast of the plurality of images.
 7. The apparatus according to claim4, wherein the at least one processor further executes instructions to:arrange a plurality of blocks in the region in each of the plurality ofimages; and detect a feature amount from each of the plurality of blocksin one image of the plurality of images.
 8. The apparatus according toclaim 7, wherein the at least one processor further executesinstructions to arrange a plurality of blocks similar to the pluralityof blocks in each of the plurality of blocks.
 9. The apparatus accordingto claim 8, wherein the at least one processor further executesinstructions to re-arrange the plurality of blocks based on a focusingarea.
 10. The apparatus according to claim 9, wherein the at least oneprocessor further executes instructions to re-arrange the plurality ofblocks with at least one of a changed number of blocks, a changed sizeof each of the blocks, and changes positions of the blocks.
 11. Theapparatus according to claim 7, wherein the predetermined condition is acondition where a number of the motion vectors detected is greater thana predetermined number.
 12. The apparatus according to claim 7, whereinthe predetermined condition is a condition where a magnitude of each ofthe motion vectors is greater than a predetermined threshold value. 13.The apparatus according to claim 7, wherein the predetermined conditionis a condition where deviation of the motion vectors is less than apredetermined threshold value.
 14. An apparatus comprising: a sensor topick up a plurality of images at different in-focus positions in anoptical axis direction; at least one memory configured to storeinstructions; and at least one processor in communication with the atleast one memory and configured to execute the instructions to: arrangea region in each of the plurality of images; detect a feature amountfrom the region in each of the plurality of images; and align theplurality of images by using the feature amount, and in a case where thedetected feature amount does not satisfy a predetermined condition,re-arrange the region.
 15. A method comprising: arranging a region ineach of a plurality of images different in in-focus position in anoptical axis direction; detecting a feature amount from the region ineach of the plurality of images; and aligning the plurality of images byusing the feature amount, and in a case where the detected featureamount does not satisfy a predetermined condition, re-arranging theregion.
 16. A non-transitory computer-readable storage medium whichstores a program for causing a computer of an apparatus to execute amethod, the method comprising: arranging a region in each of a pluralityof images different in in-focus position in an optical axis direction;detecting a feature amount from the region in each of the plurality ofimages; and aligning the plurality of images by using the featureamount, and in a case where the detected feature amount does not satisfya predetermined condition, re-arranging the region.